Papers with prevent catastrophic forgetting
TADA: Efficient Task-Agnostic Domain Adaptation for Transformers (2023.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained transformer-based language models are limited in their expressiveness and domain knowledge. |
| Approach: | They propose a task-agnostic domain adaptation method which is modular, parameter-efficient, and data-efficient. |
| Outcome: | The proposed method is efficient and modular, parameter-efficient, and data-efficient. |
Composable Sparse Fine-Tuning for Cross-Lingual Transfer (2022.acl-long)
Copied to clipboard
| Challenge: | Adapters and sparse fine-tuning have been developed to improve transfer learning . a number of approaches have been proposed to improve performance of fine-untuners . |
| Approach: | They propose a method that fine-tunes the entire set of parameters of a large pretrained model . they use adapters and sparse fine-uning to improve model efficiency . |
| Outcome: | The proposed method outperforms adapters in cross-lingual transfer benchmarks. |
Emergent Communication Pretraining for Few-Shot Machine Translation (2020.coling-main)
Copied to clipboard
| Challenge: | state-of-the-art models that rely on multilingual pretrained encoders achieve sample efficiency in downstream applications, but lack abundant amounts of unlabelled text. |
| Approach: | They propose a method to pretrain neural networks via emergent communication from referential games by grounding communication on images as a crude approximation of real-world environments. |
| Outcome: | The proposed method significantly improves machine translation in few-shot settings and provides an evaluation protocol to probe the properties of emergent languages ex vitro. |
Not All Parameters Are Created Equal: Smart Isolation Boosts Fine-Tuning Performance (2025.emnlp-main)
Copied to clipboard
| Challenge: | Extensive experiments demonstrate that our approach significantly alleviates task interference and forgetting. |
| Approach: | They propose a framework for supervised fine-tuning for large language models . they first fine-tail the model on each task to identify its core parameter regions . |
| Outcome: | The proposed framework outperforms vanilla fine-tuning and baselines on multiple public benchmarks on reasoning, dialogue, instruction following, and more. |